System and method for memory hub-based expansion bus

ABSTRACT

A system memory includes a memory hub controller, a memory module accessible by the memory hub controller, and an expansion module having a processor circuit coupled to the memory module and also having access to the memory module. The memory hub controller is coupled to the memory hub through a first portion of a memory bus on which the memory requests from the memory hub controller and memory responses from the memory hub are coupled. A second portion of the memory bus couples the memory hub to the processor circuit and is used to couple memory requests from the processor circuit and memory responses provided by the memory hub to the processor circuit.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional of pending U.S. patent application Ser.No. 10/810,229, filed Mar. 25, 2004.

TECHNICAL FIELD

The present invention relates generally to a memory system for aprocessor-based computing system, and more particularly, to a hub-basedmemory system providing expansion capabilities for computer components.

BACKGROUND OF THE INVENTION

Computer systems use memory devices, such as dynamic random accessmemory (“DRAM”) devices, to store data that are accessed by a processor.These memory devices are normally used as system memory in a computersystem. In a typical computer system, the processor communicates withthe system memory through a processor bus and a memory controller. Thememory devices of the system memory, typically arranged in memorymodules having multiple memory devices, are coupled through a memory busto the memory controller. The processor issues a memory request, whichincludes a memory command, such as a read command, and an addressdesignating the location from which data or instructions are to be read.The memory controller uses the command and address to generateappropriate command signals as well as row and column addresses, whichare applied to the system memory through the memory bus. In response tothe commands and addresses, data are transferred between the systemmemory and the processor. The memory controller is often part of asystem controller, which also includes bus bridge circuitry for couplingthe processor bus to an expansion bus, such as a PCI bus.

In memory systems, high data bandwidth is desirable. Generally,bandwidth limitations are not related to the memory controllers sincethe memory controllers sequence data to and from the system memory asfast as the memory devices allow. One approach that has been taken toincrease bandwidth is to increase the speed of the memory data buscoupling the memory controller to the memory devices. Thus, the sameamount of information can be moved over the memory data bus in lesstime. However, despite increasing memory data bus speeds, acorresponding increase in bandwidth does not result. One reason for thenon-linear relationship between data bus speed and bandwidth is thehardware limitations within the memory devices themselves. That is, thememory controller has to schedule all memory commands to the memorydevices such that the hardware limitations are honored. Although thesehardware limitations can be reduced to some degree through the design ofthe memory device, a compromise must be made because reducing thehardware limitations typically adds cost, power, and/or size to thememory devices, all of which are undesirable alternatives. Thus, giventhese constraints, although it is easy for memory devices to move“well-behaved” traffic at ever increasing rates, for example, sequeltraffic to the same page of a memory device, it is much more difficultfor the memory devices to resolve “badly-behaved traffic,” such asbouncing between different pages or banks of the memory device. As aresult, the increase in memory data bus bandwidth does not yield acorresponding increase in information bandwidth.

In addition to the limited bandwidth between processors and memorydevices, the performance of computer systems is also limited by latencyproblems that increase the time required to read data from system memorydevices. More specifically, when a memory device read command is coupledto a system memory device, such as a synchronous DRAM (“SDRAM”) device,the read data are output from the SDRAM device only after a delay ofseveral clock periods. Therefore, although SDRAM devices cansynchronously output burst data at a high data rate, the delay ininitially providing the data can significantly slow the operating speedof a computer system using such SDRAM devices. Increasing the memorydata bus speed can be used to help alleviate the latency issue. However,as with bandwidth, the increase in memory data bus speeds do not yield alinear reduction of latency, for essentially the same reasons previouslydiscussed.

Although increasing memory data bus speed has, to some degree, beensuccessful in increasing bandwidth and reducing latency, other issuesare raised by this approach. For example, as the speed of the memorydata bus increases, loading on the memory bus needs to be decreased inorder to maintain signal integrity since traditionally, there has onlybeen wire between the memory controller and the memory slots into whichthe memory modules are plugged. Several approaches have been taken toaddress the memory bus loading issue. For example, reducing the numberof memory slots to limit the number of memory modules that contribute tothe loading of the memory bus, adding buffer circuits on a memory modulein order to provide sufficient fanout of control signals to the memorydevices on the memory module, and providing multiple memory deviceinterfaces on the memory module since there are too few memory moduleconnectors on a single memory device interface. The effectiveness ofthese conventional approaches are, however, limited. A reason why thesetechniques were used in the past is that it was cost-effective to do so.However, when only one memory module can be plugged in per interface, itbecomes too costly to add a separate memory interface for each memoryslot. In other words, it pushes the system controllers package out ofthe commodity range and into the boutique range, thereby, greatly addingcost.

One recent approach that allows for increased memory data bus speed in acost effective manner is the use of multiple memory devices coupled tothe processor through a memory hub. A computer system 100 shown in FIG.1 uses a memory hub architecture. The computer system 100 includes aprocessor 104 for performing various computing functions, such asexecuting specific software to perform specific calculations or tasks.The processor 104 includes a processor bus 106 that normally includes anaddress bus, a control bus, and a data bus. The processor bus 106 istypically coupled to cache memory 108, which, is typically static randomaccess memory (“SRAM”). Finally, the processor bus 106 is coupled to asystem controller 110, which is also sometimes referred to as a busbridge. The system controller 110 serves as a communications path to theprocessor 104 for a variety of other components. For example, as shownin FIG. 1, the system controller 110 includes a graphics port that istypically coupled to a graphics controller 112, which is, in turn,coupled to a video terminal 114. The system controller 110 is alsocoupled to one or more input devices 118, such as a keyboard or a mouse,to allow an operator to interface with the computer system 100.Typically, the computer system 100 also includes one or more outputdevices 120, such as a printer, coupled to the processor 104 through thesystem controller 110. One or more data storage devices 124 are alsotypically coupled to the processor 104 through the system controller 110to allow the processor 104 to store data or retrieve data from internalor external storage media (not shown). Examples of typical storagedevices 124 include hard and floppy disks, tape cassettes, and compactdisk read-only memories (CD-ROMs).

The system controller 110 includes a memory hub controller 128 that iscoupled to the processor 104. The system controller 110 is furthercoupled over a high speed bi-directional or unidirectional systemcontroller/hub interface 134 to several memory modules 130 a–n. As shownin FIG. 1, the controller/hub interface 134 includes a downstream bus154 and an upstream bus 156 which are used to couple data, address,and/or control signals away from or toward, respectively, the memory hubcontroller 128. Typically, the memory modules 130 a–n are coupled in apoint-to-point or daisy chain architecture such that the memory modules130 a–n are connected one to another in series. Thus, the systemcontroller 110 is coupled to a first memory module 130 a, with the firstmemory module 130 a connected to a second memory module 130 b, and thesecond memory module 130 b coupled to a third memory module 130 c, andso on in a daisy chain fashion. Each memory module 130 a–n includes amemory hub 140 that is coupled to the system controller/hub interface134, and is further coupled a number of memory devices 148 throughcommand, address and data buses, collectively shown as local memory bus150. The memory hub 140 efficiently routes memory requests and responsesbetween the memory hub controller 128 and the memory devices 148.

The memory devices 148 on the memory modules 130 a–n are typicallycapable of operating at high clock frequencies in order to facilitatethe relatively high speed operation of the overall memory system.Consequently, computer systems employing this architecture can also usethe high-speed system controller/hub interface 134 to complement thehigh clock speeds of the memory devices 148. Additionally, with a memoryhub based system, signal integrity can be maintained on the systemcontroller/hub interface 134 since the signals are typically transmittedthrough multiple memory hubs 140 to and from the memory hub controller128. Moreover, this architecture also provides for easy expansion of thesystem memory without concern for degradation in signal quality as morememory modules are added, such as occurs in conventional memory busarchitectures.

Although the memory hub architecture shown in FIG. 1 provides improvedmemory system performance, the advantages my not directly benefit thevarious components of the computer system 100. As previously described,the components, such as the graphics controller 112, the input andoutput devices 118, 120, and the data storage 124 are coupled to thesystem controller 110. It is through the system controller 110 that thecomponents 112, 118, 120, 124 access the memory modules 130 a–n. As aresult of the memory requests necessarily being coupled through thesystem controller 110, a “bottleneck” can often result since the systemcontroller 110 can handle only a finite number of memory requests, andcorresponding memory responses from the memory modules 130 a–n, at agiven time. The graphics port through which the graphics controller 112is coupled to the system controller 110 provides some relief to thebottleneck issue, since the graphics port typically provides directmemory access (DMA) to the memory modules 130 a–n, as well known in theart. That is, the graphics controller 112 is able to access the memorymodules 130 a–n directly, with limited intervention by the systemcontroller 110.

As well known, arbitration schemes are implemented by the systemcontroller 110 in order to prioritize memory requests it receives fromthe various components 112, 118, 120, 124, as well as memory requestsreceived from the processor 104. The arbitration schemes that areimplemented attempt to provide efficient memory access to the variouscomponents 112, 118, 120, 124, and processor 104 in order to maximizeprocessing capabilities. Some memory requests are given priority overothers regardless of the order in which the requests are received by thesystem controller 110, for example, the processor 104 is often givenhighest priority to access the memory modules 130 a–n to avoid thesituation where processing is halted while the processor 104 is waitingfor a memory request to be serviced. As sophisticated as arbitrationtechniques have become, it is still unlikely that bottlenecks at thesystem controller 110 can be completely avoided. Even where a componentis given direct memory access to the memory modules 130 a–n, such as thegraphics controller 112, it is nevertheless subject to the arbitrationroutine that is implemented by the system controller 110, andconsequently, the component does not have unlimited access privileges tothe memory modules 130 a–n. It is by the nature of the architecture usedin the computer system 100, namely, providing access to the memorymodules 130 a–n through the single point of the system controller 110,that makes bottlenecks at the system controller 110 inevitable.Therefore, there is a need for an alternative system and method forproviding components of a processing system, such as a computer system,access to memory resources.

SUMMARY OF THE INVENTION

A system memory in one aspect of the invention includes a memory hubcontroller, a memory module accessible by the memory hub controller, andan expansion module coupled to the memory module having a processorcircuit also having access to the memory module. The memory hubcontroller provides memory requests to access memory devices, and thememory module includes a plurality of memory devices coupled to a memoryhub. The memory hub receives the memory requests, accesses the memorydevices according to the memory requests, and provides memory responsesin response to the memory requests. The processor circuit of theexpansion module provides memory requests to the memory hub of thememory module to access the memory devices, and processes data returnedin the memory responses from the memory hub. The memory hub controlleris coupled to the memory hub through a first portion of a memory bus onwhich the memory requests and the memory responses are coupled. A secondportion of the memory bus couples the memory hub to the processorcircuit and is used to couple memory requests from the processor circuitand memory responses provided by the memory hub to the processorcircuit.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a partial block diagram of a conventional processor-basedcomputing system having a memory hub-based system memory.

FIG. 2 is a partial block diagram of a processor-based computing systemhaving a memory hub-based memory system according to an embodiment ofthe present invention providing peripheral component expansioncapabilities.

FIG. 3 is a partial block diagram of a memory hub of the hub-basedmemory system of FIG. 2.

FIG. 4 is a partial block diagram of a processor-based computing systemhaving a memory hub-based memory system according to an alternativeembodiment of the present invention providing peripheral componentexpansion capabilities.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 2 illustrates a processor based computing system 200 according toan embodiment of the present invention. The system 200 includes many ofthe same functional blocks as previously described with reference toFIG. 1. As such, the same reference numbers will be used in FIG. 2 as inFIG. 1 to refer to the same functional blocks where appropriate. Thesystem 200 includes a processor 104 coupled to a system controller 110through a processor bus 106. As in FIG. 1, the processor performsvarious computing functions, for example, executing software to performspecific calculations or tasks, and the processor bus 106 typicallyincludes an address bus, a control bus, and a data bus. A cache memory108 is also coupled to the processor bus 106 to provide the processor104 with temporary storage of frequently used data and instructions. Aspreviously discussed with respect to FIG. 1, the system controller 110serves as a communications path to the processor 104 for a variety ofother components. Typically, this includes one or more input devices118, such as a keyboard or a mouse, to allow an operator to interfacewith the system 200, one or more output devices 120, such as a printer,and one or more data storage devices 124 to allow the processor 104 tostore data or retrieve data from internal or external storage media (notshown).

As shown in FIG. 2, the system controller 110 includes a memory hubcontroller 128 to which several memory modules 130 a–c are coupled overa high speed bi-directional or unidirectional system controller/hubinterface 134. The controller/hub interface 134 includes a downstreambus 154 and an upstream bus 156 which are used to couple data, address,and/or control signals away from or toward, respectively, the memory hubcontroller 128. As shown in FIG. 2, the memory modules 130 a–c arecoupled in a point-to-point architecture such that the memory modules130 a–c are connected one to another in series. Each memory module 130a–c in the system 200 includes a memory hub 240 that is coupled to thesystem controller/hub interface 134, and is further coupled a number ofmemory devices 148 through command, address and data buses, collectivelyshown as bus 150. As previously mentioned, the memory hub 240efficiently routes and arbitrates memory requests and responses betweenthe memory hub controller 128 and the memory devices 148. As will beexplained in further detail below, the memory hub 240 can receive memoryrequests and provide memory responses in both downstream and upstreamdirections over the downstream and upstream buses 154, 156,respectively.

In contrast to the computer system 100 of FIG. 1, the system 200includes a component expansion module 230 coupled to the controller/hubinterface 134. As shown in FIG. 2, the component expansion module 230includes a graphics controller 234 coupled to local memory devices 248over a local graphics/memory bus 250. The graphics controller 234, thelocal graphics/memory bus 250, and the local memory devices 248 can beof conventional design and operation, as well known in the art. Thegraphics/memory bus 250 includes command, data, and address buses aswell known in the art. A video bus 260 can be used for coupling videodata from the graphics controller 234 to a video terminal (not shown) asknown in the art. It will be appreciated that the component expansionmodule 230 replaces the graphics controller 112 of the computer system100. That is, the component expansion module 230 can provide thecomputer graphics capabilities and functionality of the graphicscontroller 112.

Although the component expansion module 230 is shown in FIG. 2 as havinglocal memory devices 248, access to data stored in the system memory,such as memory modules 130 a–c, is often required for processing by thegraphics controller 234. For example, the memory provided by the localmemory devices 248 may not be sufficient to store all of the graphicsdata necessary for rendering a scene. As a result, the bulk of thegraphics data is typically loaded into system memory, with the graphicscontroller 234 retrieving the portion of graphics data necessary forrendering the current scene from the system memory. Additionally, sinceaccess to the local memory devices 248 is typically limited to thegraphics controller 234, data that has been first processed elsewhere,for example, by the processor 104, must be stored to a location in thesystem memory for retrieval by the graphics controller 234 before beingstored in the local memory devices 248 for further processing. Thus,access to the memory modules 130 a–c by the component expansion module230 is often necessary.

The arrangement of the system 200 allows for access to the memorymodules 130 a–c by the component expansion module 230 withoutintervention by the system controller 110. As previously discussed, thememory hubs 240 can receive memory requests and provide memory responsesin both the downstream and upstream directions. By adopting a consistentcommunication protocol with the memory hubs 240 of the memory modules130 a–c, communication with the memory hubs 240 of the memory modules130 a–c can be performed directly by the component expansion module 230,thereby eliminating the need for intervention by the system controller110. As a result, access to the memory modules 130 a–c is not limited togoing through the system controller 110, but the component expansionmodule 230 can access the memory modules 130 a–c directly. In contrast,the graphics controller 112 in the computer system 100 (FIG. 1) istypically coupled to the system controller 110 through an advancedgraphics port, and although the graphics controller 112 has DMA accessto the memory, it is still nevertheless subject to the memory requestand memory response loading issues of the system controller 110. In thesystem 200, however, the graphics controller 234 is not subject to theloading issues of the system controller 110.

Many suitable communication protocols are known in the art, includingthe use of command packets that include appropriate information formaking memory requests to particular memory modules 130 a–c in thesystem 200 and providing memory responses in return. For example,command packets can include information such as identification data foruniquely identifying the particular memory request, address informationfor identifying a particular memory module 130 a–c to which the memoryrequest is directed, and memory device command information, includingmemory addresses, command type, and where a write operation isrequested, data can be included as well. Other protocols can be used aswell, and it will be appreciated by those ordinarily skilled in the artthat the present invention is not limited by the particular protocolimplemented.

Additionally, the arrangement of the system 200 reduces the memoryrequest and response load on the system controller 110 since it isrelieved from handling the memory requests from a requesting entity,namely the graphics controller 112 (FIG. 1). For these reasons, thelikelihood that a memory request and response bottleneck occurring atthe system controller 110 is also reduced. Moreover, by coupling thecomponent expansion module 230 to the controller/hub interface 134rather than to the system controller 110, the number of buses in thesystem 200 can be reduced.

FIG. 3 illustrates a portion of the memory hub 240 (FIG. 2). The memoryhub 240 includes four link interfaces 302, 304, 306, 308 coupled to across bar switch 310 by respective local link buses 312, 314, 316, 318.Memory controllers 324 a, 324 b are further coupled to the cross barswitch 310 through respective local memory controller buses 326 a, 326b. The cross bar switch 310, which may be of a conventional orhereinafter developed design, can couple any of the link interfaces 302,304, 306, 308 to each other. The link interfaces 302, 304, 306, 308 maybe either unidirectional or duplex interfaces, and the nature of thememory accesses coupled to or from the link interfaces 302, 304, 306,308 may vary as desired, including communication protocols havingconventional memory address, control and data signals, shared addressand control signals and packetized memory access signals. As shown inFIG. 3, the link interfaces 302 and 304 are coupled to the downstreambus 154 and the link interfaces 306 and 308 are coupled to the upstreambus 156.

The cross bar switch 310 can also couple any of the link interfaces 302,304, 306, 308 to either or both of the memory controllers 324 a, 324 b,each of which is coupled to a plurality of memory devices 148 (not shownin FIG. 3) over respective local memory buses 150 (FIG. 2). The memorycontrollers 324 a, 324 b may be conventional memory controllers or somehereinafter developed design for a memory controller. The specificstructure and operation of the memory controllers 324 a, 324 b will, ofcourse, depend on the nature of the memory devices 148 used in thememory modules 130 a–c. The cross bar switch 310 couples the linkinterfaces 302, 304, 306, 308 to the memory controllers 324 a, 324 b toallow any of a plurality of memory access devices to write data to orread data from the memory devices 148 coupled to the memory controllers324 a, 324 b. The cross bar switch 310 further couples the linkinterfaces 302, 304, 306, 308 to the memory controllers 324 a, 324 b toallow any data to be transferred to or from the memory devices 148coupled to the memory controllers 324 a–324 b from or to, respectively,other memory modules 130 a–c containing a memory hub 240. Thus, aspreviously discussed, the memory hub 240 is capable of receiving memoryrequests and providing memory responses in both downstream and upstreamdirections over the downstream and upstream buses 154, 156.

It will be appreciated by those ordinarily skilled in the art that FIG.3 illustrates merely a portion of the memory hub 240, and that thememory hub 240 will generally include components in addition to thoseshown in FIG. 3. For example, a cache memory for each of the memorycontrollers 324 a, 324 b can be included for storing recently orfrequently accessed data retrieved from or stored in the memory devices148. Additionally, a write buffer can also be included for accumulatingwrite addresses and data directed to the memory devices 148 serviced bya respective one of the memory controllers 324 a, 324 b if the memorydevices 148 are busy servicing a read memory request or other readrequests are pending. Such components are conventional and known in theart. These components have been omitted from FIG. 3 in the interest ofbrevity and clarity. It will further be appreciated by those ordinarilyskilled in the art that in some applications, components shown in FIG. 3may be omitted. For example, although the memory hub 240 shown in FIG. 3includes two memory controllers 324 a, 324 b the number of memorycontrollers may vary as desired.

FIG. 4 illustrates a processor-based computing system 400 according toanother embodiment of the present invention. The system 400 includesmany of the same functional blocks as previously described withreference to FIGS. 1 and 2. As such, the same reference numbers will beused in FIG. 4 as in FIGS. 1 and 2 to refer to the same functionalblocks where appropriate. The system 400 includes a processor 104coupled to a memory hub controller 428 through a processor bus 106. Acache memory 108 is also coupled to the processor bus 106 to provide theprocessor 104 with temporary storage of frequently used data andinstructions. The memory hub controller 428 is further coupled to asystem controller 110, which serves as a communications path to theprocessor 104 for a variety of other components. As shown in FIG. 4,data storage device 124 is coupled to the system controller 110 to allowthe processor 104 to store data or retrieve data from internal orexternal storage media (not shown).

The memory hub controller 428 is coupled over a high speedbi-directional or unidirectional system controller/hub interface 134 toseveral memory modules 130 a–c. The controller/hub interface 134includes a downstream bus 154 and an upstream bus 156 which are used tocouple data, address, and/or control signals away from or toward,respectively, the memory hub controller 428. Each memory module 130 a–cin the system 400 includes a memory hub 240 that is coupled to thesystem controller/hub interface 134, and which is further coupled anumber of memory devices 148 through command, address and data buses,collectively shown as bus 150. The memory hub 240 efficiently routesmemory requests and responses between the memory hub controller 128 andthe memory devices 148. As with the memory hub 240 shown in FIG. 2,memory requests and memory responses can be provided in both downstreamand upstream directions over the downstream and upstream buses 154, 156,respectively, by the memory hub 240.

Coupled in series with the memory modules 130 a–c over the downstreamand upstream buses 154, 156 are component expansion modules 230 and 430.The component expansion module 230, as previously described withreference to FIG. 2, includes a graphics controller 234 coupled to localmemory devices 248 over a local graphics/memory bus 250. The componentexpansion module 230 provides video data over a video bus 260 to a videoterminal (not shown), as known in the art. In contrast to the system 200of FIG. 2, the system 400 further includes the component expansionmodule 430. The component expansion module 430 includes an input/output(IO) processor 434 coupled to local memory devices 448 over a localmemory device bus 450. Although the component expansion module 430includes local memory devices 448, the IO processor 434 has access tosystem memory, for example, memory modules 130 a–c, as well.

Unlike the systems 100 and 200, where the input and output devices 118,120 are coupled to the system controller 110, input and output devices(not shown in FIG. 4) can be coupled to the system 400 through thecomponent expansion module 430 and a high-speed IO bus 460. By includingthe component expansion module 430, memory request and response loadingon the system controller 410 can be reduced compared to theconfiguration of systems 100 and 200. Using a consistent communicationprotocol with the memory hub 240 over the downstream and upstream buses154, 156, the memory hub controller 428, the IO processor 434, and thegraphics controller 234, can each access the memory modules 130 a–cindependently. As shown in FIG. 4, the memory modules 130 a–c and thecomponent expansion modules 230, 430 are series coupled in anarrangement that takes advantage of the point-to-point architectureprovided by the downstream and upstream buses 154, 156. The memory hubcontroller 428, the IO processor 434 and the graphics controller 234each have a respective memory module 130 a–c which can be used primarilyfor servicing memory requests by the respective component. That is, thememory module 130 a can be used primarily by the memory hub controller428 for servicing memory requests from the processor 104 and the systemcontroller 410, the memory module 130 b can be used primarily by thecomponent expansion module 430 for servicing memory requests from the IOprocessor 434, and the memory module 130 c can be used primarily by thecomponent expansion module 230 for servicing memory requests from thegraphics controller 234. Thus, although the memory hub controller 428,the component expansion module 430, and the component expansion module230 have access to any of the memory modules 130 a–c, memory requestsfrom each of the requesting entities can be primarily serviced by arespective memory module 130 a–c. As a result, the memory request andresponse loading that is conventionally handled by the system controller110 is distributed throughout the memory system, thereby reducing thelikelihood of memory requests and response being bottlenecked throughone access point.

It will be appreciated by those ordinarily skilled in the art that theembodiments shown in FIGS. 2 and 4 have been provided by way of example,and are not intended to limit the scope of the present invention.Modifications can be made to the previously described embodimentswithout departing from the scope of the present invention. For example,the system 400 has been described as providing each of the requestingcomponents, the memory hub controller 428, the component expansionmodule 430, and the component expansion module 230, with a respectivememory module 130 a–c for primarily servicing memory requests. However,only portions of the memory available on a memory module 130 a–c can beused for one requesting entity, with the remaining memory of the samememory module 130 a–c allocated for primarily servicing the memoryrequests of another requesting entity. That is, the allocation of memoryis not limited to a per module basis, but can be allocated as desired.Additionally, the order in which the memory modules 130 a–c and therequesting entities are coupled, namely the memory hub controller 428,the component expansion module 430, and the component expansion module230, can be changed and remain within the scope of the presentinvention. Although the order of the requesting entities can be arrangedadvantageously with respect to the memory modules 130 a–c, as previouslydescribed with respect to having a primary memory for servicing memoryrequests, the present invention is not limited to any specific order ofcoupling of the memory modules and requesting entities.

From the foregoing it will be appreciated that, although specificembodiments of the invention have been described herein for purposes ofillustration, various modifications may be made without deviating fromthe spirit and scope of the invention. Accordingly, the invention is notlimited except as by the appended claims.

1. A method of accessing a system memory having first and second memory modules, each memory module having a plurality of memory devices and a respective memory hub coupled to a respective plurality of memory devices, the memory hubs coupled to each other, the method comprising: generating memory requests by a memory hub controller for access to the plurality of memory devices of the first memory module; coupling the memory requests from the memory hub controller to the first memory hub; generating memory requests by a processor circuit of an expansion module coupled to the second memory hub for access to the plurality of memory devices of the second memory module; coupling the memory requests from the processor circuit to the second memory hub; generating memory responses by the second memory hub in response to receiving the memory request from the processor circuit; and coupling the memory responses from the second memory hub to the processor circuit of the expansion module.
 2. The method of claim 1 wherein coupling the memory requests from the processor circuit to the second memory hub comprises coupling the memory requests to a unidirectional upstream bus and coupling the memory responses from the second memory hub to the processor circuit comprises coupling the memory requests to a unidirectional downstream bus.
 3. The method of claim 1 wherein generating memory responses by the second memory hub comprises: coupling the memory request to a memory controller coupled to the plurality of memory devices of the second memory module; translating the memory request into memory device signals coupled to the plurality of memory devices; retrieving data from the memory devices in response to the memory device signals; and generating the memory response including the retrieved data. 